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Motion-Compensated Television Coding: Part I 

By A. N. NETRAVALI and J. D. ROBBINS 
(Manuscript received August 29, 1 978) 

We present methods of estimating displacements of moving objects 
from one frame to the next in a television scene and using such 
displacements for frame-to-frame prediction. Displacement is esti- 
mated by a recursive algorithm which seeks to minimize a functional 
of the prediction error. Several simplifications of the algorithm are 
presented which make it attractive for hardware implementation. 
Performance of the algorithm is evaluated by computer simulations 
on two sequences of moving images containing various amounts and 
types of motion. In both cases, the use of displacement- based (or 
motion- compensated) prediction results in bit rates that are 22 to 50 
percent lower than those obtained by simple "frame-difference" pre- 
diction, which is used commonly in the interframe coders. 

I. INTRODUCTION 

Television signals are generated by scanning a scene several times 
a second even though there may not be any change in the scene from 
one frame to the next. This results in a considerable frame-to-frame 
redundancy in the signal. Existence of this redundancy has long been 
recognized, and several measurements have been made to quantify it. 
However, the first real demonstration of a frame-to-frame coder which 
used redundancy between successive frames was made in 1969 by 
Mounts. 1 Since then, several improvements have been made to the 
basic frame-to-frame encoder resulting in prototypes or real imple- 
mentations of coder-decoder pairs. 2 " 5 These are the subjects of two 
excellent surveys, 6,7 the first one covering material up to 1972 and the 
second one up to 1978. As is evident from these works, most frame-to- 
frame coders are based on the following: 

( i) Segmenting each television frame into two parts, one part which 
is the same as the previous frame and the other part (called the moving 
area) which has changed from the previous frame. 

(ii) Transmitting two types of moving area information: (a) ad- 
dresses specifying the location of the picture elements in the moving 
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area, and (b) information by which the intensities of the moving area 
elements can be updated. 

(Hi) Matching the coder bit rate to the channel rate. Since the 
motion in a real television scene occurs randomly and in bursts, the 
amount of information about the moving area will change as a function 
of time. To transmit it over a constant bit rate channel; (a) smooth 
out the transmitted information rate by storing it in a buffer prior to 
transmission, and (b) use the buffer fullness to regulate the encoded 
bit rate by varying amplitude, spatial, and temporal resolution of the 
television signal. 

Intensities of the moving area picture elements are transmitted by 
predictive coding which sends frame difference, element difference, or 
line difference (or their combination) as the differential signal. At- 
tempts have been made to optimize the picture quality within the 
constraints of the buffer size and the channel rate. 

Simultaneously with these implementations, computer simulations 
have been used to explore other improvements of the frame-to-frame 
coders. It has long been recognized 8,9 that, if an estimate of the 
translation of an object is obtained, more efficient predictive encoding 
can be performed by taking differences of elements in the moving 
areas with respect to elements in the previous frame that are appro- 
priately translated. We refer to such schemes as "Motion-Compensated 
Coding Schemes." Their success depends obviously on the following: 
(i) the amount of purely translational motion of objects in a real 
television scene,* (ii) the ability of an algorithm to estimate the 
translation with an accuracy that is desirable for good prediction of 
intensity, and (Hi) robustness of the displacement estimation algo- 
rithm when amplitude, spatial, and temporal resolution of the trans- 
mitted picture are lowered due to buffer fullness. 

Several methods of estimating the displacement of an object in a 
television scene have been proposed. Methods of point-by-point cor- 
relation 9 or pattern matching used in scene analysis 10-12 appear to be 
too complex for present-day implementation, especially if displacement 
needs to be defined with resolution finer than the sampled grid of a 
television frame. Simpler displacement estimation techniques 1314 uti- 
lize the relation between the spatial differential and temporal differ- 
ential signals. They would be easier to implement. Another approach 
is adaptive linear prediction using elements in the previous frame (or 
field) which are displaced in all directions (horizontal left, right; vertical 
up, down) by a certain maximum amount and adapting the coefficients 
to minimize an intensity error function. 15 All these techniques assume 



* So that the television camera sees the objects in pure translation, it is also necessary 
that the lighting be uniform in the camera field of view. 
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that the displacement is constant within a block of picture elements. 
This assumption presents difficulties in scenes with multiple moving 
objects, occluding objects, as well as different parts of the same object 
moving with different displacements. Of course, decreasing the size of 
the block makes this assumption more realistic, but then the quality 
of displacement estimate suffers. 

The techniques proposed by Haskell 15 do not require an explicit 
estimate of translational displacement and are applicable even if the 
motion is not precisely translational. However, they also work on 
blocks of picture elements and require a matrix inversion in addition 
to other complicated operations and, therefore, do not appear to be 
easily implementable without a significant simplification. 

We present several new techniques for estimating motion. They 
attempt to minimize recursively a measure of the motion-compensated 
prediction error. Thus, given an tth estimate of displacement, we 
obtain the (i + l)th estimate such that, in general, the motion- 
compensated prediction error resulting from ( i + l)th estimate is lower 
than that using the ith estimate. The recursive minimization is per- 
formed by a gradient or steepest descent algorithm. Considerable 
freedom exists in the specific choice of this algorithm. Our choices 
have been guided primarily by a desire to implement this algorithm in 
real time. 

In Section II, we present a derivation of several motion-estimation 
algorithms and describe some simulations to evaluate their perform- 
ance. Section III contains modifications of one of the algorithms of 
Section II for use in a frame-to-frame coder. Here we evaluate the 
performance of the algorithm in the context of a frame-to-frame coder. 
Section IV contains a discussion and the conclusions of our study. 
Many enhancements of the algorithm are possible, and some of these 
will be presented in a companion paper. 16 

II. MOTION ESTIMATION 

In this section, we derive some simple algorithms for estimating 
motion. They attempt to minimize recursively a certain quantity 
(function of the motion estimation error). If the changes in successive 
television frames are due to translation of an object, then the algorithm 
iterates in a gradient or steepest descent direction such that the 
consecutive estimates converge to an estimate of translation. A proof 
of convergence of such a scheme under certain assumptions is given in 
the appendix and is supported by a large number of computer simu- 
lations presented at the end of this section. 

2. 1 Motion estimation in a block of pels 

We mentioned in the introduction that most algorithms in the 
literature for estimating translation of an object from a television scene 
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assume that the translation is constant within a block of picture 
elements (pels). We start with one such algorithm developed by Limb 
and Murphy 13 and Cafforio and Rocca 14 and show how it can be 
modified to obtain a better estimate of translation. This is done 
primarily to define a quantity which is fundamental to our recursive 
algorithm introduced in the next section. 

Assuming a 2:1 interlaced raster format, let /(x, t - t) and /(x, t) be 
the intensity values of the two successive frames as a function of 
spatial location x (a two-dimensional vector) and time t. The time 
between the two frames is t. If an object moves in translation, then in 
the moving area (disregarding the uncovered background): 

/(x, t) = /(x - D, t - t), (1) 

where D is the translation vector of the object during the time interval 
[t - t, t\ The frame difference at spatial position x is given by 

FDIF{±) = /(x, t) - 7(x, t - t) 

= 7(x, t) - J(x + D, t), (2) 

which can be written, for small D, by Taylor's expansion about x as 

FDIF(x) = - D T VJ(x, t) + Higher Order Terms in D, (3) 

where V is the gradient with respect to x and superscript T on a vector 
or matrix denotes its transpose. If the translation of the object is 
constant over the entire moving area (except for the uncovered back- 
ground) and if the higher order terms in D can be neglected, then both 
sides of the above equation can be summed over the entire moving 
area to obtain a good estimate for translation. We recognize that VJ 
can be taken to be a vector of element and line differences (EDIF, 
LDIF) if the intensity is available on a discrete grid as is the case in 
most coding situations. Using linear regression, we get t), an estimate 
of D as: 



t> = - 



l V/(x, 0-V/(x, t)' 



X FDIF(x)-VI(x,t) 



]■ 



which can be written as 

i -i 



£> = - 



£ EDIFHx) I EDIF (x)- LDIF (x) 

|_£ EDIF (*)■ LDIF (x) %LDIF 2 (x) 



^FDIF(x)-EDIF(x) 
L^FDIFW-LDIFix)! 
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where all the summations are over the entire moving area. This can be 
approximated* by assuming that 



and then 



fl-- 



£ EDIF(x)>LDIF(x)=0 

moving 
area 



£ FDIF(x).EDIF(x)/Y t EDIF 2 ix) 
L£ FDIF(x).LDIF(x)/Z LDIF 2 (x)j 



(4) 



This can be further approximated* by avoiding the multiplications in 
the sums as: 



D-- 



l FDIF(x)sign(EDIF(x))' 
l\EDIF(x)\ 

£ FDIF (x)sign(LDIF (x)) 

. l\LDIF(x)\ . 



(5) 



where 



sign(z) = 



0, if 2 = 

z 



(6) 



A 



. otherwise. 



This algorithm is identical to one of the algorithms given by Limb and 
Murphy. 13 Its accuracy may be improvedf by several modifications 
suggested by Limb and Murphy 13 and Cafforio and Rocca. 14 

We suggest another modification which improves the above motion 
estimator further. First, we note that the above estimates are good as 
long as D is small. As D increases, the quality of the approximation 
becomes poor. This can be overcome to some extent by linearizing the 
intensity function around an initial estimate of D. This is possible in 
a television situation, where there is an estimate of D for every field. 
Thus, for the ith field, displacement estimate T)' can be obtained by 
linearizing the intensity function around the displacement estimate for 
the (i — l)th field. This process results in the following recursion: 



D' = fl'" 1 + U', 



(7) 



* We make no attempts to justify these two approximations. They are made merely 
to derive the algorithm given in Ref. 13. Perhaps, instead of the least squares, some 
other criterion may lead to the same result (i.e., eq. (5)) with fewer assumptions. 

f The improvement suggested by Limb (Ref. 13) was simulated for comparison 
purposes. However, for the type of pictures used in Section 3.3, the performance did not 
change noticeably by incorporating the improvement. 
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where IV" 1 is an initial estimate of tV and U' is the update of fr' -1 to 
make it more accurate, i.e., an estimate of D — D' _1 . 

We now define the quantity dfd (x, ft' -1 ), called the displaced frame 
difference, which is analogous to FDIF(x) used in (4) and (5), 

dfd(x, fi'- 1 ) = J(x, t) - /(x - fi'-\ t - t) (8) 

dfd is defined in terms of two quantities: (i) the spatial location x at 
which it is evaluated and (ii) the displacement ft 1-1 with which it is 
evaluated. Obviously, in the case of a two-dimensional grid of discrete 
samples, an interpolation process would be used to evaluate /(x — 
b*"\ t - t) for nonintegral values of t)'~\ As defined, dfd has the 
property of converging to zero as t)' converges to the actual displace- 
ment, D, of the image. Following the same steps as were used in the 
derivation of eq. (5), we get: 

dfd(x, ft'" 1 ) = /(x, t) - /(x + D - f> L \ t) 

= -(D - ft 1 '" 1 ) T - VJ(x, t) + Higher Order Terms. (9) 

Neglecting higher order terms and making approximations similar to 
the above results in an estimate of D — 6'" 1 which, when combined 
with eq. (7), yields: 

'£ dfd(x, D , " 1 )sign(^Z)/F(x))" 



& = & - 1 - 



Z\EDIF(x)\ 
X dfd(x, D'- 1 )sign(LZ>/F(x)) 



(10) 



%\LDIF(x)\ 

where the summations are again carried over the entire moving area. 
This may be simplified slightly by noting that if the initial estimate 
of displacement ft' -1 has only integral components, then dfd( • , • ) can 
be computed without interpolation. Let [D] denote an integer approx- 
imation to D. This can be obtained either by truncating or rounding 
both the components of the vector D. 

I dfd(x, [l^'- 1 ]).sign( J Bi)/F(x))' 



D' = [D 11 ] - 



Z\EDIF<*)\ 
£ dfd(x, [D'- 1 ]).sign(LZ)/F(x)) 



(11) 



£|LZ)/F(x)| 

We describe the performance of both the above algorithms later in 
this section. 

2.2 Pel-recursive estimation of motion 

We mentioned in the introduction that there is an advantage in 
recursive algorithms which iterate on a pel-by-pel (or on a small block 
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of pels) basis, i.e., they revise their displacement estimate at every 
moving area pel. Such recursive algorithms overcome, to a large extent, 
the problems of multiple moving objects, as well as different parts of 
an object undergoing different displacements, provided the recursion 
has sufficiently rapid convergence. Since we intend to use the displace- 
ment estimator for predictive coding, our algorithm should in some 
manner attempt to minimize the resulting prediction error. Also, since 
the prediction error is calculated for transmission anyway, its use in 
the recursive estimation of displacement does not result in extra 
computations and is therefore advantageous. Thus, if a pel at location 
Xa is predicted with displacement ft' -1 and intensity I(x a — ft' -1 , t — 
t), resulting in prediction error DFD(Xa, ft'" 1 ), the estimator should try 
to produce a new estimate, ft', such that|DFD(x a , ft')|<| DFD(x a , 
ft ,-1 )|. To this end, we attempt to recursively minimize [dfd(x, ft)] 2 at 
each moving area element using a gradient type of approach. For 
example, 

G' = ft'"' - ( € /2)V D [DFD(x a ,ft'- 1 )] 2 

-ft , - 1 -€DFD(Xa,ft , - , )V D DFD(Xa,ft'~ 1 ), 

where V D is the gradient with respect to displacement D and c is a 
positive scalar constant. The gradient V D may be evaluated using the 
definition of dfd and noting that 

V D (DFD(x a , ft" 1 )) - + V/(x a - ft'" 1 , t - T), (12) 

where V is the gradient with respect to x. This gives us 

ft' = ft - 1 - €DFD(x a , ft'-W/fca - ft '-', t - t), (13) 

where dfd and V7 are evaluated by interpolation for nonintegral D ,_1 . 
A significant reduction in computation of V7 is achieved by quantizing 
ft' -1 to an integral value. Thus, if [ft' _l ] represents a rounded or 
truncated value of each of the components of ft' -1 , then the estimator 
of eq. (13) can be simplified to 

ft' = ft'"' - €DFD(X a , ft ,_1 )V/(X a - [ft'" 1 ], t - t). (14) 

It should be pointed out that V c could have been evaluated using (9), 
resulting in an estimator in which V7 is evaluated at (x a , t) instead of 
(x a - ft' -1 , t - t) as above. This second method implies an assumption 
regarding the expansion of /( • , • ), which may not be valid if D — ft' -1 
is large. Also, there is no difference in the computational complexity if 
it is assumed that a linear interpolation of 7(x, t — r) is used to 
compute dfd, and the resulting displaced line and element differences 
are used to define VZ of eq. (14). 

It is interesting to observe that at every iteration we add to our old 
estimate a vector quantity parallel to the direction of the spatial 
gradient of image intensity and whose magnitude is proportional to 
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the motion-compensated prediction error. It may be seen from eq. (9) 
that if the displacement error (D - ft'" 1 ) is orthogonal to the intensity 
gradient VZ, the displaced frame difference dfd (• , •) is zero, giving a 
zero update for recursion of eq. (14). This may happen even though 
the object may have actually moved. However, this is not a failure of 
the algorithm, but rather is identical to the situation in which an 
intensity ramp is translated and only motion parallel to the ramp 
direction (V7) is perceived. Motion perpendicular to the ramp direction 
is unobservable, and as such is arbitrary. It is only through the 
occurrence of edges with differing orientations in real television scenes 
that convergence of ft' to actual D is possible. 

The quantities involved in the above algorithm are shown in Fig. 1. 
An initial estimate of the displacement at pel x a , D 1-1 , is to be updated 
using (14), yielding 6'. Using the initial estimate ft' -1 and x„, the 
samples in the previous frame in the neighborhood of spatial position 
x a - f>' _1 are located (for example: samples b, c, d, e, and f). The 



FIELD j-1 f- 
• I 



PRESENT / 
FIELDj / 
/ 



TIME 




/KXa-D' - ' .t-T) 



l(Xa.t) * 



Kx.t) 



Fig. 1 — Recursive motion estimation. Displacement estimate D' ' is updated at pel a. 
Gradient of intensity, V/(x„ - 0/""'], t - t), is obtained by using intensities at pels b, c, 
d, e, f in the field [j - 2). 
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samples of this neighborhood are then used to compute the update 
term in conjunction with /(Xa, t). Thus, in Fig. 1, the components of 
intensity gradient may be approximated by 

EDIF=(I e -I c )/2 (15) 

LDIF=(I b -I f )/2. (16) 

Similarly, I(xa - D' -1 , t - t) may be approximated by a two- 
dimensional linear interpolation using the intensities of the neighbor- 
hood. In the next section, we present several simplifications of this 
basic algorithm and adapt it for frame-to-frame coding. Recursive 
algorithms, using a model of the video process for motion estimation 
in a different context, are described in Ref. 17. 

2.3 Motion estimator performance 

In this section, we give simulation results for the first two motion 
estimators, algorithm I (eq. (5)) and algorithm II (eq. (11)). Obviously, 
the performance depends upon the type of scene and, even then, it is 
not clear how to measure the performance. We have used two types of 
scenes. The first is produced synthetically using the following formula: 



/(x, t) = 



127 if ||R||>100 

(17) 
127 (1 + e-^cosfO^.TT. ||R||), otherwise, 



where R = x - (x„ + DO, 127 is the background intensity (on a scale 
of to 255), x is the location of the center of the moving pattern at 
t = 0, D is the displacement of the pattern per frame (i.e., velocity), 
and | • | denotes the Euclidean norm. This formula produces a series of 
alternating light and dark concentric rings with exponentially decreas- 
ing radial intensity variation. This pattern was chosen because it 
contains a distribution of edges with different direction and height. 
J(x, t) was sampled both in time t and space x to produce a set of four 
frame sequences with strictly horizontal translation ranging from 0.5 
to 6.0 pels per frame. 

The second type of scene is also a collection of four frame sequences 
containing an object approximately in horizontal translation. These 
are the same as shown in Fig. lb of Ref. 15. They contain a mannequin's 
head which was moved at various nominal speeds from 0.4 to 4.7 pels 
per frame. 

The first type of scene was chosen so that an exact velocity was 
known and, therefore, the performance of motion estimators could be 
evaluated rather easily by measuring the deviation from this known 
velocity. A second measure of performance was obtained by computing 
the "match entropy" of the elements in the moving area of each field. 
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That is, using the displacement estimate tV, obtained from two con- 
secutive fields, the entropy of dfd(x, IV) was computed using a linear 
two-dimensional interpolation over the moving area elements of frame 
/(x, t). This quantity is similar to the entropy of the prediction error; 
however, some future information is used in its calculation. Due to the 
presence of shadows and nonuniform illumination, the second type of 
scene does not possess a precisely defined velocity, and therefore only 
match entropy was used to evaluate the motion estimators for this 

type of scene. 

In all the simulations of the results of algorithms I and II of this 
section and the next section, the moving area elements were deter- 
mined by an algorithm similar to that described in Ref. 13. Likewise, 
the definition of EDIF{x) and LDIF(x) used in the simulations of 
algorithm I may be found in Ref. 13. For EDIF(x), this involved the 
averaging of the element differences at I(x, t) and J(x, t - t). LDIF (x) 
was computed in a similar manner. To extend this definition for use in 
algorithm II, the corresponding differences at /(x, t) and /(x -0J " ], 
t — t) were averaged. 

The performance of algorithms I and II is given in Figs. 2a and 2b 
for the synthetic scenes. As seen in Fig. 2a, the error in the estimated 
velocity* using algorithm II is considerably smaller than that of 
algorithm I, especially at higher velocities. The peak observed in the 
estimates of algorithm I is perhaps due to the insensitivity of the 
algorithms to a per-frame shift equal to the period (10.0 units) of the 
synthetic moving image. The initial estimate of displacement for 
algorithm II was assumed to be 0. It is interesting to note that the 
curve of algorithm I is approximately the same as that of algorithm II 
after the first iteration; but after only three iterations, algorithm II 
converges to its curve in Fig. 2a. In Fig. 2b, the match entropies 
resulting from the estimates of Fig. 2a are shown. Again, the superiority 
of algorithm II over algorithm I is clearly seen. Also for comparison, 
we have included the entropies of FDIF(t) of picture elements in the 
moving area. As expected, both algorithms I and II show significant 
improvements compared to frame differences. These conclusions for 
synthetic pictures remained generally unchanged when the direction 
of motion was changed and when random frame-to-frame noise of -40 
dB (s/n ratio) was added to the scenes. 

Performance of algorithms I and II for the second type of scene is 
given in Fig. 3. Here, since we do not have available an exact velocity, 
we only give the match entropy at the various nominal horizontal 
velocities used to create the scenes. While both algorithms result in 
match entropies considerably smaller than the frame difference en- 



* Average of last four out of six estimates. 
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(a) 



ALGORITHM 
WITHOUT NOISE- 
WITH NOISE- 




12 3 4 5 6 

ACTUAL HORIZONTAL VELOCITY IN PELS PER FRAME 




17 3 4 5 6 

ACTUAL HORIZONTAL VELOCITY IN PELS PER FRAME 



Fig. 2 — Performance of two motion estimators which obtain one displacement esti- 
mate per field on synthetic pictures. Algorithm II results in considerable improvement 
over algorithm I. Zero entropy indicates that, in addition to a perfect estimate, there 
was no interpolation error in evaluating the prediction error. Estimated vertical velocity 
components (in lines/frame) were ± 0.3 for algorithm I and ± 0.02 for algorithm II. 

tropy, we see that the performance of the two appears about the same. 
This result is perhaps due to the averaging of the displacement 
components over a large block (i.e., the moving area of an entire field). 
It should be mentioned that the performance of both the algorithms 
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4 - 



1 2 3 4 5 

NOMINAL HORIZONTAL VELOCITY IN PELS PER FRAME 

Fig. 3— Performance of two motion estimators which obtain one displacement esti- 
mate per field on moving mannequin. 

can be improved by separating the uncovered background, as suggested 
by Cafforio and Rocca. 14 

III. CODER PERFORMANCE USING RECURSIVE DISPLACEMENT 
ESTIMATION 

In the previous section, we developed motion estimators and dis- 
cussed the performance of estimators which obtain one velocity esti- 
mate per field. We used scenes which had only one object, moving with 
a nearly uniform translation^ displacement. However, such scenes are 
unrealistic and, therefore, estimators which can dynamically adjust to 
motion of objects are more desirable. In this section, we first describe 
our recursive estimator in more detail and then evaluate its perform- 
ance on scenes that are much more realistic. We then modify our basic 
estimator so that it can be incorporated in a frame-to-frame encoder 
and evaluate the coder performance. We note that we have paid little 
attention to a very important facet of frame-to-frame coders: resolution 
control using the contents of the buffer. It is important that motion 
estimation does not suffer immensely in lower resolution modes of the 
coder. Although some of these issues will be considered in part II, 16 
more realistic performance evaluation can only be done using a hard- 
ware coder working on real scenes. 

The first scene, called Judy, consists of 64 frames (2:1 interlaced 
fields) of 256 X 256 samples each, obtained at 30 times a second and 
sampled at Nyquist rate from a video signal of 1 MHz bandwidth, and 
contains head-and-shoulders view of a person engaged in a rather 
active conversation. The portion of a frame classified as moving area 
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varies from 15 to 51 percent. Also, the motion is not translational, and 
different parts of the scene move differently (such as lips, eyes, and 
head). Four frames of this scene are shown in Fig. 4. 

The second scene, called Mike and Nadine, consists of 64 frames 
with the same resolution as the scene Judy and contains a panned full 
body view of two people briskly walking around each other on a set 
with severe nonuniform illumination. The percentage of a frame clas- 
sified as moving area varied from 92 to 96 percent. Four frames from 
this sequence are shown in Fig. 5. 

3. 1 Basic estimator performance 

The basic recursive estimator consists of the following: the displace- 
ment estimate is updated at each moving area picture element using 
eq. (14), i.e., 

D ' S=D '" 1 -i^- DFD to» 1>, '" I >' v/ fc ■-[A , '- , L'-t), (18) 

where l}' is the estimated displacement of moving area pel x,, iV -1 is 
the last estimate formed prior to pel x, during the pel-by-pel, line-by- 
line (interlaced raster scan order) iteration through the moving area, 
VZ is the spatial gradient of the intensity, approximated by EDIF and 
LDIF defined in (15) and (16), and [D' _1 ] denotes rounded f)' 1 . A two- 
dimensional linear interpolation is used to evaluate dfd (interpolation 
is discussed in detail in Section 3.3.3). Instead of using the previous 
frame for the intensity 7(x, t — t), we have used the previous field. 
Relative advantages of this choice are discussed later. Also, both the 
horizontal and vertical components in the displacement error estimate 
(i.e., the second term of the right-hand side of eq. (18)) are clipped at 
a magnitude of ( 8 /i2s), so that the displacement from pel to pel is not 
allowed to change by more than Vie pel/field and Vie line/field. This 
avoids the possibility of rapid oscillations in displacement due to noise. 
The accuracy used in computation of displacement and interpolation 
is V128 pel (or line) per field. 

In all the simulations of this section, with the exception of the results 
of algorithm I appearing in Fig. 7, the moving area picture elements 
were determined by the following rule: pel z (Fig. 6) is classified as a 
moving area pel if either 

(i)\FDIF(z)\>T 2 

or 

(ii)\FDIF(z)\>T 1 

and 

\FDIF\ata, b, c, or d > T, 

and 

\FDIF\atA, B, C, or D > T lt 
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Fig. 6— Configuration of pels used in the moving area segmentor. 

where T x and T 2 are two thresholds, and T 2 2* T\ in most cases. This 
segmenter is quite similar to the one in Refs. 4 and 15. It overcomes, 
to a large extent, the effects of frame-to-frame noise which otherwise 
produce a large number of isolated moving area elements. For the 
basic estimator, the moving area was segmented with thresholds T\ 
= 4 and T 2 = 255 (on a scale of to 255). 

The performance of this basic estimator is given in Fig. 7 for the 
sequence Judy. Also for comparison, we have included the entropies of 
the moving area frame differences and the match entropy of algorithm 
I of Section II. On the average, match entropy for the recursive 
estimator was lower by about 1.4 bits/moving area pel than that of the 
frame differences, while algorithm I only resulted in approximately 
half of this decrease.* 

3.2 Basic coder performance 

In order to incorporate the estimator as a part of a coder, several 
other choices have to be made. In this section, we describe a basic 
coder and its performance. The next section contains modifications 
and simplifications of the basic coder and their effects on the perform- 
ance. The basic coder consists of the following: 

3.2. 1 Displacement estimator 

In the predictive coder, since the displacement estimate is not 
transmitted to the receiver, it has to be derived from the previously 
encoded and transmitted data. We derive a displacement estimate for 
a previously transmitted pel and use it for computation of the predic- 



* Results in Fig. 7 for algorithm I were obtained by using a slightly different moving 
area segmenter given in Ref. 13. 
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tion of the present pel. The two natural choices for the relative location 
of the displacement estimate are the pel above and the pel to the left 
of the predicted pel. We have chosen the previous line (i.e., the pel 
above in the same field), since it relieves hardware timing constraints 
that would have otherwise resulted. If the previous line pel is not a 
moving area pel, we use the initial displacement estimate that would 
have been used by the above pel if it were moving. Such use of 
displacement estimates for prediction is shown in Fig. 8. We see that 
several elements (e.g., elements j + \J + 2, • • • J + 4 of the present 
line) may be predicted with the same displacement estimate if the 
corresponding elements of the previous line are not moving area 
elements. Other details of the estimator remain the same as those 
given in Section 3.1 for the basic estimator. 

3.2.2 Segmentation 

Every field was first divided into two segments: moving area and 
background, using the segmentation strategy described in Section 3.1. 
For a good picture quality, the thresholds 7\ and T 2 were chosen to be 
1 and 3 (on a scale of to 255), respectively. Moving area was further 
divided into two segments: (i) compensable regions, where motion- 
compensated prediction is adequate and, therefore, no update infor- 
mation need be sent; and (ii) uncompensable regions which require an 
update (i.e., transmission of prediction error). This segmentation of 
the moving area was performed using the following rule: 

(i) If the magnitude of the motion-compensated prediction error 
for a moving area pel is greater than 3 (out of 255), then it is 
called uncompensable. 
(ii) To reduce the occurrence of isolated compensable pels, com- 
pensable pel runs of length 1, 2, and 3 between uncompensable 
pels were bridged by calling them uncompensable. 
This segmentation is shown in Fig. 9. 




PRESENT LINE 



O - BACKGROUND ELEMENT Q - MOVING ELEMENT 

Fig. 8— Use of displacement estimates (D') in prediction process. Displacement 
estimates are formed at every moving area pel in the same order as the scanning process. 
Displacement estimate of the previous line element is used for prediction of the present 
line picture elements. 
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-o- 

PRESENT LINE 



O BACKGROUND ELEMENTS 

D - COMPENSABLE MOVING AREA ELEMENTS 

■ - UNCOMPENSABLE MOVING AREA ELEMENTS 



Fig. 9 — Motion-compensated coder segmentation. Elements whose motion-compen- 
sated prediction error is lower than a threshold are called compensable and are not 
transmitted. 



DECISION LEVELS 1.7.13.21.30,39.49,60.71.82,95,108,121,134,147,160,173,255 



REPRESENTATIVE LEVELS 0,4,10,17,26,35.44,55,66,77,87.102,115,128,141,154,167,179 

Fig. 10 — Quantizer for basic coder. Only the positive side of the symmetric quantizer 
is shown. 



3.2.3 Quantization 

The prediction error was quantized using a 35-level symmetric 
quantizer shown in Fig. 10. Other quantizers were also investigated. 
The relative performance of the coder did not depend heavily on the 
quantizer. However, the picture quality degraded as would be expected 
by using a very coarse quantizer. The quantizer of Fig. 10 was chosen 
since it gave good picture quality, although the quantization error was 
clearly visible to a trained eye. 

The performance of this basic coder was evaluated by computing 
the following quantities: (i) the entropy of the prediction errors of 
uncompensable pels, and the entropy of the run lengths defining the 
(ii) background pels, (Hi) compensable pels, and (iv) uncompensable 
pels. The last three quantities are the addressing costs associated with 
not sending the prediction errors for certain pels. Figure 1 1 shows the 
overall bit rate associated with frame difference conditional replenish- 
ment encoding and motion- compensated encoding for both of our test 
sequences. Conditional replenishment encoding was done using the 
quantizer of Fig. 10 and moving area segmentation similar to that of 
the motion-compensated coder. The quality of pictures for both cases 
appeared to be approximately the same. Two characteristics of the 
motion-compensated coding are worth noting: (i) the average bits/ 
field is reduced by about 44 percent; (u) more importantly, during the 
times when large amounts of frame difference data are generated due 
to either rapid movement or a high percentage of moving area, the 
motion-compensated coder does significantly better. This type of be- 
havior may help to decrease the occurrence of temporal overload 
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associated with most conditional replenishment coders. Part of the 
reason for such a significant reduction in bit rate due to movement 
compensation is seen from Figs. 12 and 13 which show the results for 
conditional replenishment and motion-compensated coding, respec- 
tively. In these figures, the pel intensity is proportional to the bits 
required to code the prediction error for that pel. Uncompensable pels 
form a very small portion of the moving area and their prediction error 
is also small. For the scene Mike and Nadine, the output of each 
encoder is more than twice that obtained for Judy. This increase is 
mainly due to the panning of the camera. For this scene, intended as 
a severe test for the encoders, motion-compensated encoding resulted 
in a 22-percent reduction in average encoded bits. 

Figure 14 shows a breakdown of the bits required for transmission 
of addresses and prediction error for scene Judy. It is interesting to 
note that, on the average, for the conditional replenishment coder the 
addressing bits comprise only about 15 to 30 percent of the total 
transmitted bits, but in the movement-compensated coder, since the 
magnitude of the prediction error is decreased, the addressing bits 
account for a much larger fraction of the total bits (50 to 65 percent). 
More efficient addressing methods are therefore desirable with motion 
compensation. 

3.3 Variation in coder structure 

In this section, we simplify the coder and evaluate the effect of 
various modifications on the coder performance. 

3.3.1 Coder prediction 

It is known' 5 that, in conditional replenishment encoders, instead of 
using a frame differential prediction, line or element difference of the 
frame difference can be used with advantage. Thus in Fig. 8, for picture 
element x, +3 , the conditional replenishment coder would send 
[FDIF(x j+3 ) - FDIF(x j+2 )] as an element difference of frame differ- 
ence prediction error. Similar modifications can be made to the motion- 
compensated coder. Here we transmit line or element differences of 
the displaced frame differences. Thus for picture element x 7+3 , we 
transmit [dfd(x, +3 , D ,+3 ) - dfd(x, +2 , D ,+3 )] as an element difference 
of displaced frame difference prediction error. In cases such as pel x, 
of Fig. 8, where the previous element is not in the moving area, their 
fd as well as dfd is assumed to be zero. The performance of this 
predictor modification is given in Fig. 15. As seen from this figure, 
element differencing improves both the frame differential and motion- 
compensated coders by about 5 to 15 percent. 

3.3.2 Addressing of moving area 

In motion-compensated coding, the addressing information takes up 
a large fraction of the total transmitted bits. We have therefore used 
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some known 18 addressing techniques and evaluated their effects on the 
total bit rate. As in Ref. 18, three techniques for addressing the moving 
area were tried. In the first technique, the beginning of each moving 
area was given an absolute address (8 bits) with respect to the begin- 
ning of the scan line, and the end of moving area cluster was given a 
special flag word different from all the prediction error codes. This is 
similar to the technique used in Ref. 2. The second technique 18 was to 
address the beginning of each cluster of moving area with respect to a 
similar cluster in the previous line as long as the magnitude of this 
differential address was less than ±8; otherwise, the cluster was ad- 
dressed with respect to the beginning of the line. Maximum differential 
address of 8 was chosen after experiments with 2, 4, 16, and 32. The 
final technique is to address all the clusters as one-dimensional run 
lengths, as in our previous sections. The performance of these three 
addressing schemes for both the conditional replenishment and mo- 
tion-compensated coders is given in Fig. 16 for sequence Judy. As 
expected, absolute addressing requires more bits than either differen- 
tial or run-length addressing. Run-length addressing is the most effi- 
cient and decreases the overall bit rate by about 10 percent compared 
to the absolute addressing in a motion-compensated coder. 

It should be noted that, to precisely evaluate the addressing cost 
associated with motion-compensated coding, it is not adequate to use 
only the run-length entropies of the three types of segments. Knowing 
the type of the last run, the receiver must be told, in some fashion, 
which of the two possible types the next run is. This information has 
been included in the simulations of Fig. 16. Comparing the motion- 
compensated coder using run-length addressing for moving area as in 
Fig. 16 with the results shown in Fig. 11 which does not include the 
next segment type of information, we see that this information ac- 
counts for about 8 percent of the bits. 

3.3.3 Interpolation 

The motion-compensated coder discussed above uses previous field 
intensities for interpolation rather than previous frame intensities. Use 
of previous field has two advantages: (i) the displacement values are 
generally smaller, since fields occur at every l/60th of a second, 
whereas frames occur at l/30th of a second, and (ii) in a hardware 
coder, since only a certain number of elements from the previous field 
or frame would be made available for interpolation with the same 
number of adjacent available elements, almost twice the amount of 
motion can be accommodated. On the other hand, one disadvantage of 
using the previous field intensities is that, for a perfectly horizontal 
motion, intensities have to be obtained from the previous field by 
interpolation (due to the interlace) which introduces error in the 
computation of dfd. Another factor, which appears to be a disadvan- 
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tage but which can be overcome, is the use of an alternate field 
dropping coder mode. To relieve a buffer overflow condition caused by 
a peak in coder output, it is advantageous to drop alternate fields as is 
done in conditional replenishment coders, 4,5 however; the previous 
field is then no longer available for motion estimation. In any case.we 
experimented with both the previous field and frame intensities and 
found that the use of previous field intensities resulted in entropies of 
unquantized moving area prediction errors which were 5 to 15 percent 
lower than those of previous frame intensities. 

Of the many interpolation techniques possible, we restricted our 
attention to linear interpolation since our goal was a coder that could 
be implemented in real time. Thus, given the four surrounding pels as 
in Fig. 17, if the displacement D is written as the sum of an integral 
part D 7 , and a fractional part D f with component magnitudes D { and 
Di, then the intensity can be interpolated by a standard two-dimen- 
sional linear interpolation as: 

/ = (1 - Z)f>)[(l - D{)I D + D{I C ] 

+ Dl[(l-D{)I B +D{I A ], (19) 

where / is the interpolated intensity. It should be noted that eq. (19) 
has been used for all simulations previously described in this paper. 
Since this interpolation formula requires a large number of time- 
consuming multiplications, we tried another approach. We first select 
the three nearest (in the sense of the Euclidean norm) neighboring 
pels out of the four (e.g., pels B, C, and D in Fig. 17). This may be 
done by rounding D to form D 7 , thus locating the nearest neighbor, 
pel D, which is taken to be the corner of the three, then testing the 
sign of the rounding errors to locate the other two pels. The following 
formula is then used for interpolation: 

/= I D + D{(I C - Id) + Di(I B - Id). (20) 
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Fig. 17 — Two-dimensional linear interpolation. Displacement D is decomposed into 
integral part D' and fractional part D'. 
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Simulations using sequence Judy, comparing eqs. (19) and (20), showed 
very little differences in entropy, and since the interpolation of eq. (20) 
is significantly simpler, it is more suitable for a real-time implemen- 
tation. 

The precision with which the interpolation computation is per- 
formed was also varied. In particular, we used precisions of 2~ n ; n = 1, 
2 • • • , 7 pel (or line) per field for displacement estimates, and found, 
as expected, that the average bits/field increased as the precision was 
decreased. The average increase in bits/field relative to a precision of 
Vi28 pel/field was as follows: negligible increase for precisions greater 
than Vs pel/field, 2 to 3 percent for Va pel/field, 6 to 10 percent for V* 
pel/field, and 15 to 25 percent for % pel/field. Thus, it appears that for 
interpolation, a resolution of Va pel/field does an adequate job with 
much lower complexity compared to resolution of V128 pel/field. It 
should be noted that the recursion of eq. (18) was calculated by using 
a precision which is higher than the one used for the calculation of 
interpolation. 

3.3.4 Estimator parameters 

In eq. (18), the e was taken to be V1024 and the update term was 
clipped to (Vie) pels/field. Both these quantities were varied over a 
wide range. Performance due to such variations is given in Fig. 18. 
This shows the robustness of the estimator. Of course, increase of both 
e and the clipping level to a very high value results in a noisy estimator, 
but it gives the ability to adjust quickly to rapid changes in motion. 
On the other hand, a small value of e allows us to converge to a finer 
value of displacement, thereby allowing low prediction error. A com- 
promise between these two conflicting goals appears to be the values 
of e and clipping level used in (18). 

We mentioned earlier that the recursion of eq. (18) was calculated 
at a resolution of Vi2a pel/field. Simulation results on Judy indicate 
that, when the interpolation is calculated with resolution of Va pel/ 
field, the recursion of eq. (18) does not need to be calculated at 
resolutions higher than Vie pel/field, i.e, there is no significant improve- 
ment in performance by increasing the resolution beyond Vie pel/field 
in the calculation of displacement. To make the simulations realistic, 
all the computations were performed using integer arithmetic (on the 
computer). 

Within our investigation into the effect of precision, the following 
simplification of the update term was simulated: 

£>' = D- 1 - € .sign(DFD(x„ D'- 1 ))-sign(V/(x y , [ft'- 1 ])), (21) 

where sign of a vector quantity is the vector of signs of its components. 
The sign function, defined by (6), avoids the multiplication necessary 
for computation of the update term. Thus the update of each displace 
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ment component from one picture element to the next consists of only 
three possibilities; or ±c. The performance using such a simplification 
is given in Fig. 19. It is clear from this figure that the performance in 
terms of bits/field is virtually unaffected by using the simplified update 
term of eq. (21). 

Earlier we mentioned that we computed the displacement from the 
previous line of video and used it for computation of the motion- 
compensated prediction of the present line. This was done so that the 
displacement computation would have enough time to finish in a real- 
time encoder. It is possible, at least in simulations, to use the displace- 
ment of the previous moving area element in the same line for 
computation of the motion-compensated prediction of the present pel. 
This variation improves the performance of the encoder by about 6 to 
10 percent in terms of bits/field. 

The last simplification we attempted was in the calculation of the 
intensity gradient. Our hope was to use the same element and line 
differences which would be used in the computation of the dfd, so 
that this computation could be shared. Figure 20 shows three possible 
configurations of picture elements and the corresponding intensity 
gradients that were simulated. In the first and second configuration, 
the elements are those used for the interpolation of eqs. (19) and (20), 
respectively, while configuration III replaces pel D of configuration II 
with two pels further away. Coder performance using the three inter- 
polation configurations was similar, and since the second configuration 
is the simplest, it is more suitable for hardware implementation. 

3.4 Combined effects of simplifications 

We have so far evaluated the changes in coder performance due to 
each individual simplification. In this section, we evaluate the com- 
bined effects of the following simplifications: 

(i) The moving area is specified with absolute addresses and the 
next segment-type information is transmitted as in part 2 of the 
last section. 
(ii) The sign (•) -sign (•) estimator of eq. (21) is used for displace- 
ment estimation. 
(Hi) The three-point interpolation of eq. (20) is used for intensity 
prediction and displacement estimation at a precision of Vs pel/ 
field. 
(iv) The line and element differences of interpolation eq. (20) are 

used as the intensity gradient components of eq. (21). 
As seen from the simulated performance of the coder in Fig. 21, the 
modifications have had only a minor effect on the encoded bits/field. 
The simplified basic coder represents what we believe to be a practical 
algorithm for real-time implementation. A simplified block diagram of 
this coder appears in Fig. 22. The inner loop, which comprises the 
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: PEL D IS NEAREST TO 
THE DISPLACED PEL AS 
IN EQ. (20) 



Fig. 20— Possible pel configurations for computing intensity gradient. 

displacement estimator, attempts to match interpolated intensities of 
the last field with those of the last line by adjusting the displacement. 
The outer loop uses the displacement estimates to predict the video 
input with interpolated last field intensities. Based on frame difference, 
the segmentor limits displacement estimation (switch A) and motion- 
compensated prediction (switch B) to the moving areas. The segmentor 
also controls the selection of errors for transmission with switch C. 
Hardware for controlling the coder's output and matching it to a 
constant rate channel has been omitted in this representation. How- 
ever, it has to be included for any real-time implementation. 

IV. DISCUSSIONS AND CONCLUSIONS 

We have developed recursive estimators in this paper and simplified 
them so that a real-time implementation would be possible. Some 
characteristics of our final estimator are that it needs computation of 
interpolation intensities at resolution no more than Ve pel/field (which 
can be done without multiplication). We believe that this is in the 
realm of present-day circuit speeds. Also, by the very nature of the 
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recursive estimation, we do not need the assumption of uniform 
translational motion of a single object, which appears to be necessary 
for most of the heretofore known estimates. Performance of the simple 
estimator on the sequence Judy shows that a 30-to-50 percent improve- 
ment in bits/field is possible compared to frame-difference conditional 
replenishment. We also evaluated the performance of our simple 
estimator on a more difficult scene: Mike and Nadine. Both conditional 
replenishment and motion compensation generate significantly more 
data for this scene than for Judy due to the high percentage of moving 
area pels. However, the motion-compensated coding decreases the 
transmitted bits/field by approximately 22 percent. Thus the improve- 
ment is somewhat lower for this scene. Some of the modifications 
suggested in Part II 16 improve the performance for this type of scene. 
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APPENDIX 

In this appendix, we show that the gradient algorithm of eq. (13) 
which attempts to minimize dfd(- , •) converges, under suitable con- 
ditions, to true displacement. As in Section II, let us assume that an 
object is moving with pure translation in the field of view of the 
camera. Neglecting the uncovered background and assuming a con- 
stant and uniform displacement per unit time, we get 

7(x, t) = 7(x - D, t - t), (22) 

where D is the true displacement and t is the frame time. The pel- 
recursive motion estimation algorithm of eq. (13) can be written as: 

f>' = f>> _ € DFD(x a , D'-^V/fXa - £>'-\ t - t). (23) 

Using the definition of dfd( • , • ) from eq. (8), w get 

£>' = £>'-' -eiH^t) 

- 7(x„ - D- 1 , t - t)} V/(Xa - ft' -1 , t - t). (24) 
Substituting for 7(x a , t) from (22), 

r>« = D- 1 - € {/( Xq -D,*-t) 

- 7(Xa - ty~\ t - t)} • V/(x a - D 1 " 1 , t - t). (25) 
Using Taylor's series, we can write 

7(X a - D, t - T) - 7(Xa - t)-\ t - T) 

= (t)'" 1 - D) T . V7(xa - f>'-\ t - t) 

+ higher order terms in (£)'"' - D). 
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Substituting in (25), 

f> - D' 1 - € {(fT 1 - D) T V/(x a - t)-\ t - t)} V/(Xa - t)-\ t - r) 
+ higher order terms in (f)'" 1 - D). 
= IT 1 - € {V/(x a - t> il , t - t) • V/(x a - f>-\ t - t) 1 ) (£>'-' - D) 
+ higher order terms in (D ,_1 — D). 
Subtracting D from both sides, we get 

(£>' - D) = (D' _1 - D) - e {V/(x a - £)'-', t - t) 

+ higher order terms in (f)'" 1 - D). 
Neglecting higher order terms,* for small (fr~ l - D), 

(f)' - D) 

= [J - cfV/tXa - D'- 1 , « - T).V7(x a - f)" 1 , « - t^KD'- 1 - D) 

where J is an identity matrix of appropriate size. We now take 
statistical averages of both sides, assume statistical independence of 
the two right-hand terms.f and then apply Schwartz inequality to get 

l|fr-D|| 

« || J - e {VKx. - ft' 1 , t - t) VKx. - fr\ t - t) t ) II 

• IID'^-DH (26) 

where a bar on top denotes a statistical average. This can be written 
as: 

|| & - D || < 1 1 - e\ ma x| • || 6- 1 - D ||, (27) 

where \ max is the maximum eigenvalue of the positive semidefi nite, 
symmetric matrix V/(Xa - tT\ t - r)V/(x a - W* % t - r) T . For 
convergence of the algorithm, we need 

| 1 - eAnuJ < 1, 



i.e., 

2 



>6>0. (28) 



Since the maximum eigenvalue is hard to compute, and since it is 
upper bounded by the trace, we get 



* This may be a severe assumption in certain cases. In such cases, the ability of the 
algorithm to converge may depend upon the goodness of the initial estimate D. 

t This is traditionally done in stochastic gradient algorithms. See, for example, Refs. 
19 to 21. 
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tr[VI(x a - fr- 1 , t - T)V7(x a - £>'\ t-rf} 
But since 



> € > 0. (29) 



*r[V/(x a - f>'-\ t - t)V7( Xo - D'-\ t - t) t ] 



~ ±-Y EDIF 2 + LDIFl 

N & 

we get the following condition for convergence 

9N 
T5 — >€>0, (30) 

i EDIF 2 + LDIFl 

i-l 

where the summations are carried over the entire moving area con- 
taining N pels. 
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